Mapping and display for evidence based performance assessment

ABSTRACT

A computer-implemented method for providing a user with a performance indicator score includes receiving a first transaction message that includes historical clinical-trial performance data from one or more processors at a clinical research organization and receiving a second transaction message with health records data with parameters indicative of insurance claims data. The received historical clinical-trial performance data and the prescription data is translated into an updated database. Related records within the updated database are identified and one or more key performance indicators included in the data at the updated database for a first physician are identified. A score for each of the one or more key performance indicators are calculated and a performance indicator score record for the first physician is generated based on the calculated scores for each of the one or more key performance indicators. A multi-dimensional chart for organizing and evaluating investigators is generated.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part application of and claimspriority to U.S. application Ser. No. 14/554,553, filed on Nov. 26,2014, the entire contents of which is incorporated herein by reference.

BACKGROUND

The determination of the most efficient candidates to participate in aclinical trial can be one of the most important factors for clinicaltrial organizations. The assessment of investigators, that is,physicians or doctors that participate in clinical trials, is thereforeessential.

SUMMARY

In one general aspect, a computer-implemented method for providing auser with a performance indicator score includes receiving a firsttransaction message that includes historical clinical-trial performancedata from one or more processors at a clinical research organization andreceiving a second transaction message with health records data withparameters indicative of insurance claims data. The received historicalclinical trial performance data and the health records data istranslated into an updated database. Related records within the updateddatabase are identified and one or more key performance indicatorsincluded in the data at the updated database for a first physician areidentified. A score for each of the one or more key performanceindicators are calculated and a performance indicator score record forthe first physician is generated based on the calculated scores for eachof the one or more key performance indicators.

This and other implementations can each optionally include one or moreof the following features. In another aspect, receiving a secondtransaction message with health records data includes receiving patientdata and prescriber data. In yet another aspect generating, based on thecalculated scores for each of the one or more key performanceindicators, a performance indicator score record for the first physicianincludes calculating a weighted average of the calculated scores foreach of the one or more key performance indicators. In another aspect,the weight of particular key performance indicator to the performanceindicator score is based on a therapeutic area.

In another aspect, generating, based on the calculated scores for eachof the one or more key performance indicators, a performance indicatorscore record for the first physician includes calculating a performanceindicator score based on an algorithm. In another aspect, theperformance indicator score for the first physician is presented to theuser. In yet another aspect, a performance indicator score record forthe first physician includes generating a performance indicator scorerecord based on a subset of the one or more key performance indicators.

In another aspect, the subset of the one or more key performanceindicators used to calculate the performance indicator score record forthe first physician is selected by the user. In yet another aspect, oneor more key performance indicators for a second physician are identifiedand a score for each of the one or more key performance indicators arecalculated. A performance indicator score record for the secondphysician is generated based on the calculated scores for each of theone or more key performance indicators.

In another aspect, the first physician and second physician are rankedby the associated performance indicator score and the ranked list ispresented to the user. In another aspect, the ranked list is presentedto the user based on the score for a particular key performanceindicator. In yet another implementation, receiving a first transactionmessage that includes historical clinical-trial performance data fromone or more processors at a clinical-research organization includesreceiving a data file, a stream of data or a datagram.

In a second general aspect, a computer-implemented method for organizingclinical trial data includes obtaining identities of a plurality ofinvestigators and data representing a set of attributes associated witheach of the plurality of investigators from a first data set and asecond data set, where the first data set containing proprietary dataassociated with at least one of the investigators, and the second dataset containing third-party data associated with at least one of theinvestigators. Receiving a user input indicating a subset of attributesfrom the set of attributes associated with each of the plurality ofinvestigators. Generating a multi-dimensional chart that organizes theidentities of the plurality of investigators based on the subset ofattributes and a user designation of selected dimensions to reflect twoor more of attributes from the subset of attributes. Themulti-dimensional chart includes a first dimension representing a firstattribute from the subset of attributes, a second dimension representinga second attribute from the subset of attributes, and a plurality oficons. Each icon represents an identity of one of the plurality ofinvestigators, and each icon is positioned on the multi-dimensionalchart along the first dimension according to a value of the firstattribute associated with the represented identity and along the seconddimension according to a value of the second attribute of therepresented identity. A graphical property of each icon represents avalue of a third attribute of the represented identity. Linking each ofthe plurality of icons to a selectable record in the database so thatuser interactions with ones of the plurality of icons cause one or moreattributes associated with the ones of the plurality of icons to bealtered within the database. Providing a graphical user interface (GUI)including the multi-dimensional chart and a clinical trial roster fordisplay on a computing device. Receiving a user selection of one or moreicons from the multi-dimensional chart for inclusion in a clinicaltrial, and in response to the user selection, adding identities ofinvestigators represented by the one or more selected icons to theclinical trial roster. And, storing the identities of investigatorsrepresented by the one or more selected icons in association with theclinical trial roster and the multi-dimensional chart.

This and other implementations can each optionally include one or moreof the following features. In another aspect, the method includesmodifying an attribute associated with at least one of the addedidentities in response to adding the identities of investigatorsrepresented by the one or more selected icons to the clinical trialroster, and causing a linked icon in another multi-dimensional chart tobe modified based on the attribute associated with the at least one ofthe added identities being modified. In yet another aspect, the methodincludes selecting a subset of identities of the plurality ofinvestigators, the subset of identities including only identities ofinvestigators for whom data is available in both the first and seconddata sets. In addition, generating the multi-dimensional chart includesgenerating the multi-dimensional chart to organize the identities in thesubset of identities of the plurality of investigators, where each iconof the plurality of icons represents an identity from the subset ofidentities of the plurality of investigators. In another aspect, themethod includes receiving an approval indication for the clinical trialroster, and storing the identities of investigators represented by theone or more selected icons in association with the clinical trial rosterand the multi-dimensional chart is performed in response to receivingthe approval indication.

In another aspect, the user input indicating a subset of attributesincludes the user designation, where the user designation indicates torepresent the first attribute by the first dimension of themulti-dimensional chart and the second attribute by the second dimensionof the multi-dimensional chart. In yet another aspect, the subset ofattributes includes the third attribute, and the user input indicating asubset of attributes includes the user designation, where the userdesignation indicates to represent the first attribute by the firstdimension of the multi-dimensional chart, the second attribute by thesecond dimension of the multi-dimensional chart, and the third attributeby the graphical property of each icon of the multi-dimensional chart.

In another aspect, the data representing the set of attributesassociated with each of the plurality of investigators includes raw dataand performance indicator scores. In yet another aspect, a color of eachicon represents a fourth attribute of the represented investigator. Inanother aspect, the method includes altering icons associated with theidentities of investigators that have been added to the clinical trialroster. In another aspect, the method includes providing a list ofattributes associated with an identity of an investigator when a userselection device hovers over an icon that represents the identity of theinvestigator for display.

In another aspect, the method includes receiving a user input assigningan investigator included in the clinical trial roster to aninvestigation site, and storing the assigned investigation site inassociation with the investigator's identity in the clinical trialroster. In another aspect, the method includes receiving an approvalindication for the clinical trial roster, and sending notifications toinvestigators included in the clinical trial roster in response toreceiving the approval indication, where the notifications indicate thateach investigator has been selected to participate in the clinicaltrial. Each notification can include information related to a site towhich the investigator has been assigned for performing the clinicaltrial. Each notification can include a target number of patients for theclinical trial.

In another aspect, the method includes selecting a first subset ofidentities from among the plurality of investigators based on a firstfiltering criteria. Generating the multi-dimensional chart includesgenerating the multi-dimensional chart to organize identities in thefirst subset of identities of the plurality of investigators, where eachicon of the plurality of icons represents an identity from the firstsubset of identities of the plurality of investigators. In addition, themethod can include selecting a second subset of identities from amongthe first subset of identities, based on a second filtering criteria;and refining the multi-dimensional chart to organize identities in thesecond subset of identities, wherein each icon of the plurality of iconsrepresents an identity from the second subset of identities. The firstfiltering criteria can be an investigator attribute from one of thefirst or second data set, and the second filtering criteria can be aninvestigator attribute from the other of the first or second data set.

In another aspect, the clinical trial roster can be a first clinicaltrial roster, and the method can include receiving a second clinicaltrial roster, where the second clinical trial roster is a priorgenerated roster, and where generating the multi-dimensional chartincludes generating the multi-dimensional chart to organize identitiesfrom the second clinical trial roster, with each icon of the pluralityof icons representing an identity from the second clinical trial roster.

Other implementations of the above aspects include correspondingsystems, apparatus, and computer programs, configured to perform theactions of the methods, encoded on computer storage devices. The detailsof one or more implementations of the subject matter described in thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an example of an analytical infrastructure systemimplemented in a computing system 100.

FIGS. 2-9 illustrate example user interfaces of user interaction with awebpage application of a performance indicator tool.

FIG. 10 is a flow chart of an example process for generating aperformance indicator score record.

FIGS. 11-15 illustrate example user interfaces of user interaction witha webpage application for generating a roster of investigators.

FIG. 16 is a flow chart of an example process for generating a userinterface for generating a roster of investigators.

DETAILED DESCRIPTION

This disclosure generally describes computer-implemented methods,software, and systems for determining a performance indicator score thatreflects the assessment of the performance of an investigator withinclinical trials. The performance indicator score for a particularinvestigator, or physician, may be determined based on one or more keyperformance indicators (KPIs). The computer-implemented methods, systemsand software integrate the historical clinical trial performance datasource and the IMS data source to determine the best performinginvestigators. The data is visualized through the performance assessmentapplication in the form of maps, tables, charts, and investigatorscorecards. In some implementations, the data is visualized in userinteractive multi-dimensional charts, and clinical trial rosters can becreated and evaluated through user interaction with the charts.

Clinical trials are used to determine whether a new biomedical orbehavioral interventions are safe, efficacious, and effective. Clinicaltrials may involve research studies on human subjects and are typicallyperformed by health care providers or physicians that treat patients.The health care provider or physician may ask the patient or clinicaltrial subject to answer specific questions about a particularpharmaceutical product, vaccine, dietary supplements, or treatmentregime. It is important that physicians participating in clinical trialsare effective, and gather the required data from patients within thedesired time period. It is also important that clinical trialorganizations have the ability to access a quantifiable assessment ofphysicians. Having access to the quantifiable assessment scores ofphysicians make staffing a clinical trial a more reliable process.

Typically, a large percentage of all clinical trials under-enrollpatients and in some instances, not even a single patient is screened.Contracting and monitoring unproductive clinical trial sites leads to awaste in time and revenue. Clinical trial organizations need to identifyand enroll proven clinical trial performers, and productive sites thathave access to screen a large number of patients. This will help toavoid the enrolling chronically underperforming physicians and sitesinto clinical trials. Identifying proven performers can also lead toenrolling physicians with experience in specialties that align with aparticular clinical trial, obtaining results in a timely manner, andenrolling sites that have access to a large population of patients. Inaddition, some implementations may enable more efficient use andorganization of investigator data, thereby, reducing computing resourcesrequired to both maintain and process investigator data, and tocoordinate logistics related to organizing investigators for clinicaltrials.

FIG. 1 illustrates an example analytical infrastructure systemimplemented in a computing system 100. The computing system may beimplemented as a data processing apparatus that is capable of providingthe functionality discussed herein, and may include any appropriatecombination of processors, memory, and other hardware and software thatcan receive appropriate medical data and process the data as discussedbelow. At a high-level, the illustrated example computing system 100receives various data from sources that are participants in thehealthcare system. The sources may include integrated delivery networks(IDNs) 102, patient system 104, prescriber system 106, clinical trialorganization 108, and payer system 109. The data may include consumptiondata 110, longitudinal patient data 112, reference prescriber data 114,clinical trial historical data 116, and payers insurance data 111.

FIG. 1 illustrates the process by which an analytical infrastructure isable to integrate historical data received from clinical trialorganizations and IMS health data that includes insurance claims data,patient data and prescriber data. The data sources may be combined toform a universal data set that integrates the data sources. Theintegrated data sources may be stored as an updated database. Theupdated database may be an extension to the one or more databases thatstore the health records data and/or the clinical trial data. Theupdated database may be a computational database. The updated databasemay have the processing capability to execute extensive computations andcalculations. The updated database may perform extensive calculationsand computations on the health records data and the clinical trialhistorical data sources at the updated database high processing speeds.The updated database may be an energy efficient and time efficientdatabase. The updated database may process large amounts of clinicaltrial historical data and health records data at very high speeds. Theupdated database may have the processing ability to allow thecalculations and computations carried out on the data sources at theupdated database to be quick and effective.

It is important to understand that the system may be configured topreserve patient privacy, and will not store nominative data in anaggregated database but only de-identified data.

The physician prescription data 110 may include data regardingprescriptions prescribed by physicians within an IDN. The prescriptiondata 110 may be received directly from one or more IDNs 102 andrepresent data reflecting all prescriptions for pharmaceutical productsissued by physicians within the one or more IDNs 102, includinginformation about the type of prescription used to obtain the productand the payment method used to purchase the product. As notedpreviously, this information may be sanitized and aggregated to protectpatient privacy. The prescription data may include the total revenuespent on prescriptions based on the specific drug. In someimplementations, the data may be based on the total revenue spent on aspecific drug in a specific geographic location. Though FIG. 1 shows theprescription data 110 being provided directly from the one or more IDNs102 to the computing system 100, the prescription data 110 may becollected by one or more other intermediate systems and then provided tothe computing system 100.

The longitudinal patient data 112 may include sanitized retailpatient-level data for the one or more patient systems 104. For example,the longitudinal patient data 112 may include information about retailpharmacy-sourced prescription insurance claims, retail pharmaceuticalscripts, patient electronic medical records, and/or patient profiledata. Longitudinal patient data 112 includes information about aspectsof care for the one or more patient systems 104. Though FIG. 1illustrates the longitudinal patient data 112 as being received by thecomputing system 100 directly from one or more patient systems 104, thelongitudinal patient data 112 may be collected by one or more othersystems and then provided to the computing system 100 in a manneranalogous to the similar approach discussed for prescription data 110.Moreover, the longitudinal patient data 112 may not originate from theone or more patient systems 104, but may rather be provided by one ormore prescribers/physicians with whom the patient interacts, insurancecompanies to which a patient submits insurance claims, and/or retailersat which a patient purchases a pharmaceutical product. In someimplementations, the longitudinal patient data 112 may originate fromone or more pharmaceutical companies.

The reference prescriber data 114 may include detailed data about healthcare providers and physicians. The reference prescriber data may includedetails such as the specialty of a physician, the IDN affiliation of aphysician, and/or the health insurance network that the physician may beassociated with. This data may be obtained through prescriptions writtenby the particular prescribing physician. Though FIG. 1 illustrates thereference prescriber data 114 as being received by the computing system100 directly from one or more prescriber systems 106, the referenceprescriber data 114 may be collected by one or more other systems andthen provided to the computing system 100 in a manner analogous to thesimilar approach discussed for retail consumption data 110. Moreover,the reference prescriber data 114 may not originate from the one or moreprescriber systems 106, but rather be created and/or maintained by oneor more other entities (e.g., government agencies or professionalmedical organizations) that track information about the prescribingbehavior of prescribers 106.

The clinical trial historical data 116 may include information aboutclinical trial that were conducted by clinical trial organizations inthe past. The clinical trial historical data may include the sites andthe physicians that participated in clinical trials in the past. Theclinical trial historical data may include the date of past trials, andthe run period of the trial. For each physician that participated in thetrial, the historical data may include the number of patients screenedby the physician, the length of time the physician took to enter thedata collected. The clinical trial historical data may include any otherdata that was collected during clinical trials in the past. Though FIG.1 illustrates the clinical trial historical data 116 as being receivedby the computing system 100 directly from clinical trial organization108, the clinical trial data 116 may be collected by one or more othersystems and then provided to the computing system 100 in a manneranalogous to the similar approach discussed above.

The insurance data 111 may include information about insurance companiescovering the cost of prescriptions. For example, the insurance data 111may include information about how much of a prescription's cost wascovered by the insurance company or by Medicaid. Though FIG. 1illustrates the insurance data 111 as being received by the computingsystem 100 directly from one or more payer system 109, the insurancedata 111 may be collected by one or more other systems and then providedto the computing system 100.

The various types of data just discussed, which may include prescriptiondata 110, longitudinal prescription data 112, reference prescriber data114, clinical trial historical data 116, and insurance data 111, arereceived by computing system 100 in order to derive conclusions based onthe data. As noted previously, by the time the data is received bycomputing system 100, it should have been sanitized so that the datadoes not include private or confidential information that computingsystem 100 should not able to access.

For illustrative purposes, computing system 100 will be described asincluding a data processing module 118, a statistical analysis module120, a reporting module 122, and a storage device 124. However, thecomputing system 100 may be any computing platform capable of performingthe described functions. The computing system 100 may include one ormore servers that may include hardware, software, or a combination ofboth for performing the described functions. Moreover, the dataprocessing module 118, the statistical analysis module 120, and thereporting module 122 may be implemented together or separately inhardware and/or software. Though the data processing module 118, thestatistical analysis module 120, and the reporting module 122 will bedescribed as each carrying out certain functionality, the describedfunctionality of each of these modules may be performed by one or moreother modules in conjunction with or in place of the described module.

The data processing module 118 receives and processes one or more ofprescription data 110, longitudinal patient data 112, referenceprescriber data 114, clinical trial historical data 116, and insurancedata 111. In processing the received data, the data processing module118 may filter and/or mine the prescription data 110, longitudinalpatient data 112, clinical trial historical data 114, pharmaceuticalpurchase data 116, and insurance data for specific information. The dataprocessing module 118 may filter and/or mine the received prescriptiondata 110, longitudinal patient data 112, reference prescriber data 114,clinical trial historical data 116, and insurance data 111 for specificpharmaceuticals. After processing the received prescription data 110,longitudinal patient data 112, reference prescriber data 114, clinicaltrial historical data 116, and insurance data 111, the data processingmodule 118 aggregates the processed data into patient data 126 andprescriber data 128. These groups of data may be stored in storagedevice 124.

In other implementations, the data processing module 118 may simply sortand store, in storage device 124, processed prescription data 110,longitudinal patient data 112, reference prescriber data 114, clinicaltrial historical data 116 and insurance data, the data processing module118 for later use by other modules.

The computing systems integrate the sources of data received. Thereporting module 122 prepares reports based on the integrated datasources at the computing system 100. The reports prepared by thereporting module 122 may include the performance indicator score for aparticular physician. The performance indicator score may be a weightedaverage of score for one or more key performance indicators associatedwith the physician.

Additionally, in some implementations, the reports generated may beeither dynamic or static. The reporting module 122 may generate a reportthat includes data presented in one or more static formats (e.g., achart, a graph, or a table) without providing any mechanism for alteringthe format and/or manipulating the data presented in the report. In suchan implementation, the data presentation is generated and saved withoutincorporating functionality to update the data presentation. In someimplementations, the reporting module 122 provides a static report in aPDF, spreadsheet, or XML format. Such a format generally provides anunderstanding of the reporting module 122 as textual data or avisualization, but other forms of presenting conclusions such as audio,video, or an animation are not excluded as potential results fromreporting module 122. The reporting module 122 may provide a staticreport in a PowerPoint format.

Additionally or alternatively, the reporting module 122 may generate areport that includes controls allowing a user to alter and/or manipulatethe report itself interactively. For example, the reporting system mayprovide a dynamic report in the form of an HTML document that itselfincludes controls for filtering, manipulating, and/or ordering the datadisplayed in the report. Moreover, a dynamic report may include thecapability of switching between numerous visual representations of theinformation included in the dynamic report. In some implementations, adynamic report may provide direct access as selected by a user to someor all of the patient data 126 and prescriber data 128 prepared by thedata processing module 118 and/or the statistical analysis module 120,as opposed to allowing access to only data and/or ratings included inthe report itself. In some implementations, data displayed in a dynamicreport may be linked to the underlying database records stored atcomputing system 100 such that user interaction with the report causesmodifications to associated database records.

One or more clients 140 may interface with the computing system 100 torequest and receive reports created by the reporting system. In someimplementations, the one or more clients 140 may include a web browserthat provides Internet-based access to the computing system 100. Throughthe web browser, a user of a client 140 (e.g., a clinical trial manager,a wholesaler, a retail outlet, or a prescriber) may request a static ordynamic report from the reporting system as discussed above.

There may be any number of clients 140 associated with, or external to,the example computing system 100. While the illustrated examplecomputing system 100 is shown in communication with one client 140,alternative implementations of the example computing system 100 maycommunicate with any number of clients 140 suitable to the purposes ofthe example computing system 100. Further, the term “client” and “user”may be used interchangeably as appropriate without departing from thescope of this disclosure. Moreover, while the client 140 is described interms of being used by a single user, this disclosure contemplates thatmany users may share the use of one computer, or that one user may usemultiple computers.

The illustrated client 140 is intended to encompass computing devicessuch as a desktop computer, laptop/notebook computer, wireless dataport, smartphone, personal digital assistant (PDA), tablet computingdevice, one or more processors within these devices, or any othersuitable processing device. For example, the client 140 may include acomputer that includes an input device, such as a keypad, touch screen,or other device that can accept user information, and an output devicethat conveys information associated with the operation of the computingsystem 100. The input device may be used by client 140 to provideinstructions to computing system 100 that computing system 100 canexecute to provide information requested by client 140 from the variousdata that computing system 100 receives. The analytical infrastructuremay be supported on a webpage application that a client may use to viewthe data received by the computing system at the analyticalinfrastructure.

FIG. 2 illustrates an example administrative user interface for userinteraction with an application of a performance indicator offering. Insome implementations, interface 200 may be displayed to an internal userat IMS. The internal IMS user may act as an administrative user for theperformance indicator offering. The administrative user may configurethe performance indicator score range definitions. The administrativeuser may define the start and end values that define the medium range ofthe performance indicator score. The computing systems at the analyticalinfrastructure may then determine the low and high ranges for theperformance indicator score based on the input values from theadministrative user. In some implementations, the administrative usermay define the values for the low, medium, and high performanceindicator scores. The administrative user may define the values for therange of indicator score by any suitable means. The performance scoresdisplayed to the end user, based on the data set of the filters selectedby the user, as illustrated in FIG. 4, may be based on the performanceindicator score range defined by the administrative user. For theexample illustrated in FIG. 2, the administrative user may set the startvalue for the medium performance indicator score at 1.8 on a scale of 1to 3, and may set the end value for the medium performance indicatorscore at 2.2. In some implementations, the end user may have the abilityto define the performance indicator range. The performance indicatorscore may be based on a scale of 1 to 10, 1 to 100, or any othersuitable scale.

FIG. 3 illustrates an example administrative user interface for theperformance indicator offering. As described above, an administrativeuser may be an internal user at IMS. Interface 300 may be displayed whenthe administrative user logs into a secure connection to the performanceindicator offering. The administrative user may define the keyperformance indicators to determine how each key performance indicatorcontributes to the overall performance indicator score. Thedetermination set by the administrator may be used across the one ormore clinical trial organizations and/or pharmaceutical companies thatutilize the performance indicator offering to evaluate investigators tostaff on a clinical trial. In some implementations, the end user at theclinical trial organization and/or pharmaceutical company may have theability to configure the contribution of each key performance indicatorto the overall performance indicator score for an investigator. The enduser at the clinical trial organization and/or pharmaceutical companymay have the ability to identify which key performance indicators shouldbe included to determine the overall performance indicator score for aninvestigator.

As illustrated in FIG. 3, the user may define the key performanceindicators across the entire system by selecting the system level optionfrom a drop down tab. The user may also define the key performanceindicators according to a specific therapeutic area by selecting thetherapeutic area from the therapeutic area drop down tab. The userinterface may list the key performance indicators, the performance rangeassociated with each key performance indicator, and the default valueassociated with each key performance indicator. The list of keyperformance indicators may include active studies, considered studies,total studies, completed studies, final protocol approval to siteinitiation, institutional review board approval to site initiation, siteinitiation to first patient screened, site initiation to last patientscreened, site initiation to last patient randomized, patientsrandomized, percent of target randomized, screening rate, screen failurepercent, randomization rate, dropout percent, data entry lag, time toresolve queries, protocol violations, regulatory audits, and queries.The key performance indicators may further include in other suitable keyperformance indicator that may be revealed from the data at analyticalinfrastructure. In some implementations, the administrative user maydefine the parameters for a key performance indictor based on the dataset available at the analytical infrastructure. The key performanceindicators may be grouped into one or more categories. The categoriesmay include Experience, Workload, Cycle Time, Throughput, Datamanagement, and Data Quality. The one or more key performance indicatorsmay be grouped into any suitable category.

The Workload and Experience categories may include the active studies,considered studies, total studies and completed studies key performanceindicators. These categories of key performance indicators measure theexperience of investigators in terms of the number of clinical trialstudies an investigator has participated in the past. The investigatorexperience may be gathered from the clinical trial historical data thatis received by the computing systems at the analytical infrastructure.The active studies indicator identifies the number of currently activeclinical trial studies that an investigator is currently participating.The data processing module at the analytical infrastructure may fieldand/or mine the clinical trial historical data received. The dataprocessing module may include a study as an active study when evaluatingthe performance of an investigator based on administrative set criteria.The administrative user may exclude active and historical clinical trialstudies data from the evaluation of the performance indicator score ofan investigator. The performance data from the sites for any excludedclinical trial studies may not be included in the performance analysis.In some implementations, the end user may have the ability to determinewhether data from an active clinical trial study should be included inthe evaluation of the performance indicator score of an investigator.

The considered studies indicator identifies the number of uniqueclinical trial studies for which an investigator is being considered toparticipate. The data processing module may include a clinical trialstudy when evaluating the performance of an investigator when theinvestigator is included on a roster for a clinical trial study. Thetotal studies indicator identifies the total number of clinical trialstudies that an investigator has participated. In some implementations,this indicator may include clinical trials that have already beencompleted and clinical trials that are currently active. The completedstudies indicator identifies the number of studies that an investigatorhas completed work.

The Cycle time category may include final protocol approval to siteinitiation, institutional review board approval to site initiation, siteinitiation to first patient screened, site initiation to last patientscreened, and site initiation to last patient randomized key performanceindicators. The cycle time category of key performance indicatorsmeasures how quickly an investigator was able to achieve site initiationand patient screening milestones. The final protocol approved to siteinitiation indicator measures the time between the date the finalprotocol was approved and the date the site initiated the study. Theinstitutional review board approval to site initiation indicatormeasures the time between the date of the country level institutionalreview board approval for a clinical trial and the date the siteinitiated the clinical trial. The site initiation to first patientscreened indicator measures the time between the site's initiation dateand the date when the first patient was screened for a clinical trialstudy. The site initiation to last patient screen measures the timebetween the site's initiation date and the date when the last patientwas screened at the site. The site initiation to last patient randomizedindicator measures the time between the site's initiation date and thedate when the last patient was randomized into the site. The time periodfor the key performance indicators that fall within the cycle timecategory may measure time in days, weeks, quarters, or months, or anyother suitable time period. In some implementations, the administrativeuser may set the time unit used.

The Throughput category may include patients randomized, percent oftarget randomized, screening rate, screen failure percent, randomizationrate, and dropout percent key performance indicators. The throughputcategory of key performance indicators are used to evaluate aninvestigator's ability to process numbers of patients. The patientsrandomized indicator may be used to show the patient volume bycalculating the number of patients who were randomized by aninvestigator. The percent of target randomized indicator may indicatethe investigator's ability to meet commitments by calculating thepatients who were randomized as a percentage of the original target. Thescreening rate indicator may show an investigator's patient volume bycalculating the average number of patients who were screened per siteper unit time. The screen failure percentage may measure aninvestigator's ability to screen patients by calculating the averagepercentage of patients who failed screening out of the total number ofpatients screened. The randomization rate indicator shows aninvestigator's patient volume by calculating the average number ofpatients who were randomized per site, per unit time. The dropoutpercent shows an investigator's ability to retain patients bycalculating the average percentage of patients who left a clinical trialstudy after being randomized.

The Data Management and Quality category of key performance indicatorsmay include data entry lag, time to resolve queries, protocolviolations, regulatory audits, and queries indicators. The data entrylag indicator may evaluate the average time between a patient visit anda data entry for the visit. The time to resolve queries indicator maymeasure the average time between the query and the resolution. Theprotocol violations indicator may measure an investigator's ability tofollow the protocol for a clinical trial without incurring violations.The regulatory audits indicator may evaluate how frequently theinvestigator generated regulatory audits. The queries indicator mayevaluate how frequently the investigator generated queries.

For each listed key performance indicator, the administrative user hasthe ability to configure the medium start and medium end ranges. In someimplementations, the administrative user may configure the applicationto run using the default values. In some implementations, the end usermay have the ability to configure the performance ranges for one or moreof the key performance indicators.

FIG. 4 illustrates an example user interface for user interaction withan application of a performance indicator offering. The end user may bea client 140 that accesses the web-application to the computing systems100 at the analytical infrastructure. The user may be a user at aclinical trial organization, or the user may be a representative at apharmaceutical company that is interested in staffing a clinical trial.The end user may use the performance indicator offering to identify alist of investigators, or physicians, that are ranked based on theirclinical trial performance. The end user may use the performanceindicator offering to compare one or more physicians based on one ormore sets of metrics. The performance indicator offering may compare theone or more physicians based on patient availability derived from healthinsurance claim information. Interface 400 may be displayed when a userlogs into a secure connection with the performance indicator offering.The user may log into the application by providing a user specific username and password, or any other suitable user authentication mechanism.The webpage may be specific to individual users of the application, thatis, the webpage generated is specific to the user. In someimplementations, the user may have the option to customize theinformation displayed on the web page. The performance indicatoroffering may enable the user to evaluate the performance of aninvestigator against other investigators on a worldwide, regional, andcountry wide comparison. The user may view an investigator's performancefor a particular key performance indicator, as well as, view aninvestigator's performance against the performance of otherinvestigators.

The user interface may include a list of filters. The list of filtersmay be displayed on the left panel of the user interface. The filterspanel may include a data source filter, a therapeutic area filter, anindication filter, a program filter, a study phase filter, a regionfilter, a study type filter, and a study randomizer filter. In someimplementations, the filter panel may include a subset of these filters,and in some implementations, the filter panel may include any othersuitable filters. The end user may use a drop down selection tab toindicate which filters should be activated to generate the ranked listof investigators. For example, the end user may select what countriesand/or what other geographical locations should be included in thedataset for determining the ranked list of investigators. For theexample illustrated in FIG. 4, the end user selected all the states andcities of the United States to be included. The user interfaceillustrates a map of the selected geographical location with one or morepush pins that indicate the overall scores of investigators across theselected geographical region. In some examples, a push pin may be usedto identify the geographical location of the top ranked investigatorsbased on the user selected filters. The size and color of the push pinused to identify the geographical location may be selected by the user.The user interface may also include a ranked list of investigators thatreflects the data illustrated in the map. The ranked list may includedetails for each of the one or more investigators listed. For example,the list may include the investigator name, the geographical location ofthe investigator, the generated performance indicator score for theinvestigator, the scores for each of the one or more key performanceindicators evaluated, and the experience of the investigator. In someimplementations, the investigator's details may include the investigatorcontact information, and may include the option to add an investigatorto a roster schedule. In some implementations, the user may select torank the investigators based on one more key performance indicatorsscores. For example, the user may select to rank investigators based thescore for both the screening rate and protocol violations keyperformance indicators. In another example, the user may select to rankthe investigators by an overall performance indicator score and thescore for screening rate.

The end user may have the ability to select a particular investigatorfrom the ranked list to view further details about the selectedinvestigator. In some implementations, the details may include thespecialty area of the investigator, the IDN affiliations of theinvestigator, and the list of colleagues of the selected investigatorthat are affiliated with the same IDN network. In some implementations,the details may include a list of key performance indicators that wereused by the computing systems at the analytical interface to generatethe performance indicator score for the investigator.

FIG. 5 illustrates an example user interface for user interaction withan application of a performance indicator offering. Interface 500 may bedisplayed when a user logs into a secure connection with the performanceindicator application offering. The user may log into the application byproviding a user specific user name and password, or any other suitableuser authentication mechanism. The webpage may be specific to individualusers of the application, that is, the webpage generated is specific tothe user. The user interface 500 may be displayed when the user selectsa performance tab on the task bar.

As illustrated in FIG. 5, the computing systems at the analyticalinfrastructure may display a bubble chart based on an evaluation ofinvestigators. The results may be displayed in any suitable type ofchart. In some implementations, the user interface may display theoverall score determined for each investigator as a ranked list. Theuser may have the ability to configure the results to be displayed in achart. As illustrated in interface 500, the user interface may include atask pane that includes one or more configurable dimensions. The usermay select through drop down task bars, the x-axis dimension of thechart, the y-axis dimension of the chart, the color of the bubbles usedon the chart, and the bubble size. For the example illustrated, the userselected the bubble size to indicate the number of patients randomized,the color of the bubble to indicate the overall score for aninvestigator, the x-axis to indicate the screen rate and the y-axis toindicate the screen failure percent. The user selected metrics are thenused by the computing systems at the analytical infrastructure togenerate a bubble chart. The bubble chart clearly depicts the receiveddata for each investigator and allows the user to manipulate the datathat is plotted to get a true understanding of each of the keyperformance indicators that were used to evaluate an investigator.

FIG. 6 illustrates an example user interface that may be displayed whena user selects an individual investigator from the ranked list ofinvestigators. The user interface 600 displays the investigator's scorecard that includes the one or more metrics used to determine the overallperformance indicator score for the selected investigator. Theinvestigator score card may include the score determined for each keyperformance indicator used to determine the overall performanceindicator score. The score card may include the investigator detailssuch as the investigator name, address, therapeutic area, IDNaffiliation, or any other suitable detail. The score card may alsoinclude the investigator performance relative to aggregate metrics forthe comparison group like minimum/maximum/median/mode. In someimplementations, the set of clinical trial studies used for thecomputation of scores for the one or more key performance indicators maybe dynamically altered. For example, the data gathered from one or moreclinical trial studies may be excluded from the analysis. In someimplementations, the investigator performance is compared relative toall investigators across the selected geographical area. In otherimplementations, the investigator performance is compared relative to aworldwide group of investigators. The user may have the ability toselect the comparison group using the web-based application.

FIG. 7 illustrates a user interface 700 that may be displayed when auser of the performance indicator offering selects the overlay view. Thecomputing systems at the analytical infrastructure may integrate thehistorical clinical trial data with prescription data, patient data, andinsurance claims data. As illustrated in interface 700, the computingsystems at the analytical infrastructure, may use the integrated datasources to produce a heat map. The heat map may display, for a selectedgeographical area, the patient diagnoses for a selected time frame. Insome implementations, the heat map may display the patient diagnosisdata for an entire country. The computing systems at the analyticalinfrastructure may use one or more different colors or shade of colorsto represent the one or more different patient diagnoses ranges. In someimplementations, the heat map may be displayed according to percentilerank. For the example illustrated, four patient ranges are used torepresent the 1-25^(th), 26-50^(th), 51^(st)-75^(th) and76^(th)-100^(th) percentiles. In some implementations, the patientranges may represent any suitable diagnosis percentiles. The userinterface may also include a list of diagnoses that a user may scrollthrough and select for which diagnosis the heat map should be displayedfor. For the example illustrated in FIG. 7, the user selected thediagnosis Enteritis for the period of the past three years. Thecomputing systems at the analytical infrastructure may indicate on themap, the geographical location with the highest number of patientdiagnoses. The patient diagnosis information may be granular, and maydisplay patient diagnosis data on a county by county basis.

FIG. 8 illustrates an example user interface 800 that may be displayedwhen a user of the performance indicator offering selects to view datafor a particular diagnosis within the received health records data. Insome implementations, the health records data may include healthinsurance claims data, patient data, and prescribing data. The user mayselect which data set to mine to display diagnosis data. For example,the user may select to view diagnosis data from only health insurancedata. The user may select an indication, a sub-indication, and a timeinterval for the data to be displayed. For the example illustrated, theuser selected the indication “other arthropod borne diseases” andselected the sub-indication “Lyme disease.” The computing systems at theanalytical infrastructure may generate a map to illustrate a list ofinvestigators that have made diagnoses of the user selected diagnosis.For the example illustrated in FIG. 8, the computing systems maygenerate a ranked list of investigators based on the selected diagnosis.The map illustrated may also indicate the location of the investigatorson the ranked list. The geographical location of the rankedinvestigators may be indicated on the map with marker. The marker mayalso indicate the clinical trial experience of the investigator. In someimplementations, the marker may indicate the investigators experience bythe color of the marker. The ranked list of the investigators may bedisplayed below the generated map. The ranked list of investigators maylist the name of the investigator, the city and or state of theinvestigator, the investigator specialty, the investigator clinicaltrial experience, and the number of patients the investigator diagnosedwithin the user selected time period. In some implementations, theranked list may include the number of clinical trials the investigatorparticipated within the last year, and an indication whether theinvestigator has participated in a clinical trial carried out by theclinical trial organization associated with the user.

In some implementations, the user may narrow the list of investigatorsby applying one or more different filters to the search criteria. Forexample, the user may indicate, using a filter, to have onlyinvestigators with an overall performance score over a predeterminedthreshold be displayed. In some implementations, the user may select todisplay a ranked list of investigators based on the score for aparticular key performance indicator. For example, a user may select torank the investigators based on the key performance indicator ofcompleted studies. In some implementations, the computing systems maygenerate the ranked list of investigators based on a universal data set.The universal data set may include all data sets available to thecomputing systems at the analytical infrastructure. In theseimplementations, the user may have the ability to identify theintersection of investigators from the organization's internal data setand the health records data received at the analytical infrastructure.

In some implementations, the user may select an investigator from thelist of ranked investigators and select to view the diagnosis detailsfor the investigator. The computing systems at the analyticalinfrastructure may generate a diagnosis bar chart that displays thenumber of patients diagnosed with the user selected diagnosis eachmonth. The user may have the ability to select to display the diagnosesbased on the age of the patient diagnosed. The data may be displayed fora user selected time period. For example the user may select to have thedata displayed for the last year, or last two years. The diagnosis chartmay also break down the diagnoses based on the gender of the patient.

FIG. 9 illustrates an example user interface 900 that may be displayedwhen a user of the performance indicator offering selects to create aroster. The roster may be used to list the investigators selected by theuser to participate in a clinical trial. The user may select the desiredinvestigators from the list of ranked investigators into the roster dropdown to build the list of potential participants for the clinical trialstudy. In some implementations, when the user adds an investigator tothe roster the investigator is identified on the map by a marker. Insome implementations, the computing systems at the analyticalinfrastructure may automatically generate a roster of investigators byselecting the top ranked investigators. In some implementations, when auser selects to have investigators ranked by a particular keyperformance indicator, the computing systems at the analyticalinfrastructure may generate a roster based on all investigators thatrecords include data related to the particular performance indicator. Insome implementations, the computing systems at the analyticalinfrastructure may communicate with the computing systems associatedwith investigators. In these implementations, when an investigator isplaced in the roaster of investigators, the computing systems at theanalytical infrastructure may communicate the inclusion in the roster tothe investigator.

FIG. 10 is a flow chart of the process for generating a performanceindicator score record. The computing systems at the analyticalinfrastructure receive a first transaction message that includeshistorical clinical trial performance data from one or more processorsat a clinical research organization (1002). The received data file maybe stored at a one or more databases at the computing systems at theanalytical infrastructure. The transaction message received may be adata file, a stream of data, or a datagram. The historical clinicaltrial performance data may be data collected and maintained by one ormore processors at a clinical research organization. The historical datamay include detailed data of clinical trial studies carried out in thepast by the clinical research organization. The historical clinicaltrial performance data may include the number of patients thatparticipated in a clinical trial, the start date of the clinical trial,the completion date of the clinical trial, the number of days aphysician took to response to a query, and any other suitable historicalclinical trial data collected. The historical clinical trial data may becommunicated to the computing systems at the analytical infrastructureon a periodical basis. For example, the historical data may be receivedat the computing systems of the analytical infrastructure every week,once a month, or quarterly. The historical data may be received at thecomputing systems of the analytical infrastructure at any suitableperiod. In some implementations, the one or more processors at theclinical research organization may communicate the historical data tothe computing systems on a daily basis.

The computing systems at the analytical infrastructure receive a secondtransaction message with health records data indicative of insuranceclaims data (1004). The transaction message received may be a data file,a stream of data, or a datagram. The received health records data may bestored at a one or more databases at the computing systems of theanalytical infrastructure. The health records data received may includehealth insurance claims from a larger percentage of pharmacies acrossthe United States of America. The health insurance data me be part ofIMS health data. The IMS health data may include patient data,prescriber data, and health insurance claims data that represents alarge percentage of global health care data. The health records data mayinclude medical details for patients. The health records data may beanonymized data yet may be rich in details of gender, sex, age,diagnosis, and any other suitable patient details. The computing systemsat the analytical infrastructure may receive health records data fromone or more processors associated with one or more hospitals, treatmentfacilities, prescribing facilities, Integrated Delivery Networks (IDNs),one or more patient systems, one or more prescriber systems, and one ormore payer systems.

The computing systems at the analytical infrastructure may translate thehealth records data and the received historical clinical trialperformance data into an updated database based on the receivedhistorical clinical trial performance data (1006). The health recordsdata received may be stored at one or more databases at the computingsystems at the analytical infrastructure. The received historicalclinical trial performance data may be stored at the one or moredatabases. The two data sources may be combined logically acrossdatabases or physically within the same database to form a universaldata set that integrates the data sources. The integrated data sourcesmay be stored as one or more updated databases. The updated database maybe an extension to the one or more databases that store the healthrecords data and/or the clinical trial data. The updated database may bea computational database. The updated database may have the processingcapability to execute extensive computations and calculations. Theupdated database may perform extensive calculations and computations onthe health records data and the clinical trial historical data sourcesat the updated database high processing speeds. The updated database maybe an energy efficient and time efficient database. The updated databasemay process large amounts of clinical trial historical data and healthrecords data at very high speeds. The updated database may have theprocessing ability to allow the calculations and computations carriedout on the data sources at the updated database to be quick andeffective. The computing systems at the analytical infrastructure mayidentify related record within the updated database (1008). Dataassociated with a particular investigator or physician may be identifiedin the data sources and tagged with an identifier. Data associated witha particular geographical area, diagnosis, or physician specialty may beidentified in the data sources and tagged with an identifier.

The computing systems at the analytical infrastructure may identify oneor more key performance indicators in the updated database (1010). Thedata processing module at the computing systems may field and/or minethe universal data for data that may be used as a key performanceindicator. For example, the data processing module may identify clinicaltrial data that may have recently been received at the computingsystems, with dates that align with the current dates, and may identifythe data as belonging to a currently active study. The computing systemsat the analytical infrastructure may identity currently active clinicaltrial studies as an active studies key performance indicator. The dataprocessing module may identify data for related records with one or morekey performance indicators. The related records may be related bydiagnosis, geographical location, investigator or physician, or anyother suitable relation. For example, for a particular investigator,which may be identified with an investigator tag, the data processingmodule may identify the time between the date when the final protocolwas approved and the date the site initiated to identify the finalprotocol approved to site initiation key performance indicator.

The computing systems at the analytical infrastructure may calculate ascore for each of the one or more key performance indicators (1012). Theadministrative user at the computing systems at the analyticalinfrastructure may establish a medium performance range for eachidentified key performance indicator. The data received and identifiedfor each key performance indicator may then be compared, by thestatistical analysis module, to the medium performance range establishedby the administrative user. In some implementations, the administrativeuser may indicate the start and end points of the medium performancerange for each identified key performance indicator. In otherimplementations, the administrative user may enter two values toindicate the start and end points of the medium performance range interms of percentiles. In these implementations, the administrative userprovides a ranking that is relative to other investigators. In someimplementations, the performance indicator offering may be adapted toallow an end user to establish a medium performance range for each keyperformance indicator. The statistical analysis module at the computingsystems may assign a score to the key performance indicators. In someimplementations, the score assigned to the one or more identified keyperformance indicators may be a high score of 3, a medium score of 2, orlow score of 1. In some other implementations, any other suitable scorerange may be used.

The computing systems at the analytical infrastructure may generate aperformance indicator score record (1014). In some implementations, theadministrative user may determine which of the one or more keyperformance indicators may be used to assess the performance of aninvestigator. In some implementations, the end user may have the abilityto identify which key performance indicators should be used to generatethe performance indicator score record for a particular investigator.The end user may select that the performance indicator score record begenerated based on one key performance indicator. For example, the enduser may select to have the performance indicator score based on thescore assigned to the total studies key performance indicator. In theseexamples, the computing systems at the analytical infrastructure maygenerate a performance indicator score record, and rank one or morephysicians based on the total studies key performance indicator, that isthe number of clinical trial studies the physician has participated. Insome implementations, the end user may select that the performanceindicator score record be generated based on one or more key performanceindicators. In these implementations, the end user may have the abilityto identify the one or more key performance indicators that should beused to generate the performance indicator score record. The computingsystems at the analytical infrastructure may generate the performanceindicator score record for a particular investigator based on a weightedaverage of the one or more identified key performance indicators. Theweight to each of the one or more key performance indicators may beassigned by the administrative user. In other implementations, theweight to each of the one or more key performance indicators may beassigned by the end user. In some implementations, the computing systemsat the analytical infrastructure may generate a performance indicatorscore record for a particular investigator based on an algorithm. Insome implementations, the end user may select which key performanceindicators should be included in the calculation of the performanceindicator score record for a particular investigator, or group of one ormore investigators.

In some implementations, the performance indicator score record may bebased on a therapeutic area. In these implementations, the weight of aparticular key performance indicator may be evaluated differently basedon the therapeutic area. For example, a clinical trial organization maydecide that in a Respiratory clinical trial study, a monthly screeningrate of fifteen or more patients is considered a high performance,whereas in an Oncology clinical trial study, a screening rate of five ormore patients is considered a high performance. In some implementations,the end user may select to determine the performance indicator score forone or more investigators based on one key performance indicator. Inthese implementations, the computing systems at the analyticalinfrastructure may present the score of the selected key performanceindicator as the performance indicator score record for theinvestigator.

The computing systems at the analytical infrastructure may rank one ormore investigators/physicians based on the generated performanceindicator score. The computing systems at the analytical infrastructuremay rank one or more physicians based on the user selected keyperformance indicators and may generate a ranked list. The ranked listof investigators may be displayed to the user through the performanceindicator tool application. In some implementations, the one or moreinvestigators may be ranked based on a performance indicator scoregenerated from the evaluation of one key performance indicator. In theseimplementations, the end user may select a single key performanceindicator that the one or more investigators should be ranked accordingto. In some other implementations, the one or more investigators may beranked based on a performance indicator score generated from theevaluation of one or more key performance indicators. In theseimplementations, each of the one or more key performance indicators maybe assigned a weight and the performance indicator score may begenerated based on the weighted average of the one or more keyperformance indicators evaluated. The performance indicator score may begenerated based on an algorithm that includes the one or more leyperformance indicators. In some implementations, the user may select torank the investigators both on the overall performance indicator scoreand the score for one or more key performance indicators.

FIGS. 11-14 illustrate example user interfaces of user interaction withan application (e.g., a webpage application) for generating a roster ofinvestigators. In implementations of the present disclosure, theapplication obtains data related to a plurality of potential clinicaltrial investigators (e.g., physicians) from two or more data sets. Forexample, the application can obtain data from a proprietary data setsuch as, for example, a data set owned by a user of the application(e.g., an organization or business such as a pharmaceuticalcorporation), and from a third-party data set such as, for example, adata set compiled by a third-party data source (e.g., a health careinformation service such as IMS Health).

The proprietary data set can include, but is not limited to, clinicaltrial data created and maintained by an organization from previouslyperformed clinical trials. For example, proprietary data can includelists of clinical trial investigators that the organization may havepreviously hired or researched for clinical trials. In addition, theproprietary data can include various attributes related to the listedinvestigators, for example, contact information, data from historicalclinical trials performed for the organization, and internal performancemetrics associated with the investigators (e.g., performance metricsgenerated by the organization based on historical clinical trial datafrom trials performed by the investigator for the organization).

The third-party data set can include, but is not limited to, clinicaltrial data obtained and maintained by third-party data source based ondata related to clinical trials performed by multiple organizations,patient data, insurance data, physician data, and data from otherappropriate data sources compiled by the third-party data source. Forexample, third-party data can include lists of physicians who haveconducted or are eligible to conduct clinical trials. In someimplementations, the third-party physician list includes all or most ofthe investigators know to the organization and many additionalinvestigators (e.g., physicians) eligible to perform clinical trials. Inaddition, the third-party data can include various attributes related tothe physicians, for example, contact information, professionalinformation, performance metrics based on data from historical clinicaltrials performed for multiple organizations, proximity to patients(e.g., areas of high patient density), number and type of proceduresperformed, clinical trial experience (e.g., a total number of trialsperformed by the investigator over a defined period of time),specialties, relevant publications, etc.

In some implementations, data from the proprietary data set and thethird-party data set are obtained from separate databases (e.g., aproprietary data database and a third-party database). For example, theapplication can access the proprietary data set from a userorganization's database(s), and the third-party data set from one ormore third-party database. For instance, an organization can leaseaccess to the third-party data set, and can be provided access through auser account with the third-party.

FIG. 11 illustrates an example user interface 1100 for organizingpotential investigators for inclusion in a clinical trial (similar tothat shown in FIG. 5). The computing systems at the analyticalinfrastructure may display a multi-dimensional chart (e.g., a bubblechart) in the user interface 1100. The multi-dimensional chart may be atwo-dimensional chart that represents three or more categories of dataassociated with a plurality of investigators. The multi-dimensionalchart includes a plurality of icons (e.g., bubbles, circles, squares, orother icons) each representing an identity of an investigator. Thecomputing systems at the analytical infrastructure may obtain datarepresenting a set of attributes associated with each of theinvestigators, and represent several of the attributes along dimensionsof the chart. The attributes may include, for example, identifying datafor the investigator, key performance indicator scores, and/or raw dataused to calculate key performance indicator scores. The icons representsattribute values associated with each investigator. The position of anicon along the x-axis represents a first attribute value, the positionof the icon along the y-axis represents a second attribute value, andvarious graphical properties (e.g., size, shape, color) of the icon canrepresent additional attribute values associated with the representedinvestigator. For example, as depicted in the example user interface1100, various attributes associated with each charted investigator maybe represented by the position of each icon (e.g., along the x andy-axes), the size of the icon (e.g., the bubble size), and the color orshading of each icon (e.g., the bubble color). In some implementations,the user may select a subset of attributes associated with therepresented investigators to be displayed in the chart. In the exampleshown the user has selected the attributes Patients Randomized, OverallScore, Screening Rate, and Screen Failure % to be represented in thechart.

The user may have the ability to configure the results to be displayedin the chart. As illustrated in interface 1100, the user interface mayinclude a task pane that includes one or more configurable dimensions.The user may select investigator attributes to be represented by thechart dimensions and assign them to chart dimensions through drop downtask bars. In the example shown, the user has designated Screening Rateto be represented by the x-axis dimension of the chart, Screen Failure %to be represented by the y-axis dimension of the chart, Overall Score tobe represented by the color of the icons in the chart, and PatientsRandomized to be represented by the icon size. The user selected metricsare then used by the computing systems at the analytical infrastructureto generate a multi-dimensional chart. The multi-dimensional chartclearly depicts the data for each investigator and allows the user tomanipulate the data that is plotted to get a true understanding of eachof the key performance indicators that were used to evaluate aninvestigator.

Moreover, enabling users to define how various investigator attributesare represented along dimensions of the multi-dimensional chart mayprovide users with the ability to efficiently recognize the best andworst performing investigators according to criteria for the users'particular clinical trial. For example, the icons in the lower-rightquadrant of the multi-dimensional chart shown in interface 1100represent investigators having the highest screening rate along with thelowest screen failure percentage. Thus, a user may find the bestpotential investigators for a particular clinical trial by pairingdesired investigator attributes with appropriate dimensions of themulti-dimensional chart. Similarly, the multi-dimensional chart may beused to identify potentially incorrect or problematic investigator data(e.g., outliers). For example, an outlier icon may indicate that someportion of the charted data associated with the outlier icon isincorrect, or that there is a potential problem with the investigator'swork. In addition, some implementations of interface 1100 also mayinclude crosshairs that divide the multi-dimensional chart to helpfacilitate efficient recognition of investigator performance. Inaddition, the crosshairs may include data labels that indicate theirposition within the multi-dimensional chart. For example, the verticalcrosshair line indicates that it is positioned along the x-axis at ascreening rate of 9.6. Similarly, the horizontal crosshair lineindicates that it is positioned along the y-axis at a screen failurepercentage of 50.0.

In some implementations, interface 1100 may include a user-selectablefiltering input. The filtering input may permit a user to filter theinvestigators represented in the chart based on one or more attributesassociated with the investigators. In some implementations, the filteredset of investigators may be further reduced by selecting additionalfiltering criteria in a Refine Results pane. In some implementations,the multi-dimensional chart may include zoom and/or pan features thatallow the user to focus in on tight clusters of investigator icons(e.g., as shown in the lower-left quadrant of the chart).

FIGS. 12 and 13 illustrate an example user interface 1200 for selectinginvestigators for inclusion in a clinical trial roster. The userinterface 1200 includes a roster that may be used to list theinvestigators selected by the user to participate in a clinical trial.The user may select the desired investigators from the multi-dimensionalchart. For example, in some implementations, a user may be permitted toselect one or more of icons from the multi-dimensional chart and dragthe selected icons into the roster. As shown in FIG. 13, once a selectedicon has been dragged into the roster one or more attributes associatedwith investigator represented by the selected icon may be displayed inthe roster (e.g., Investigator Name, Data Source, and Region). In someimplementations, when the user adds an investigator to the roster theappearance of the investigator's icon is modified in the chart. Themodification to the investigator's icon may include, for example,shading, highlighting, outlining, hatching, etc. For example, the iconsassociated with the selected investigators (e.g., Darrell Willings,Kaeln Pinn, Gonzalo Meados, Erik Mastro, and Corey Donahue) are outlinedin red.

In some implementations, the computing systems at the analyticalinfrastructure may modify one or more attributes associated with theselected icons in response to the icons being added to (or removed from)the roster. For example, a considered studies attribute may beincremented (or decremented) when an associated icon is added to (orremoved from) the roster. In some implementations, the computing systemsat the analytical infrastructure may communicate with the computingsystems associated with investigators. In these implementations, forexample, when an investigator is placed in the roster, the computingsystems at the analytical infrastructure may communicate theinvestigator's inclusion in the roster to the investigator (e.g., bye-mail). In some examples, the computing systems at the analyticalinfrastructure may communicate the investigator's inclusion in theroster to the investigator until the user indicates that the roster iscomplete, for example, by approving the roster. In some implementations,the roster may be approved by another user (e.g., a supervisor of theuser or administrator).

FIG. 14 illustrates an example user interface 1400 for organizingpotential investigators for inclusion in a clinical trial including aninvestigator information display 1402. In some implementations, aninvestigator information display 1402 may be presented to a user whenthe user interacts with the icons displayed in the multi-dimensionalchart. For example, if a user interacts with a particular investigator'sicon, an investigator information display including attributesassociated with the investigator (e.g., investigator Erik Mastro) may bepresented to the user. A user interaction causing the investigatordisplay to be presented may include hovering over an icon with a userselection device (e.g., a mouse or touchpad cursor) or selecting an icon(e.g., clicking an icon or touching an icon on a touchscreen).

FIG. 15 illustrates an example user interface 1500 for modifyingattributes of investigators included in a clinical trial roster. In someimplementations, an interface 1500 may be presented to a user when theuser selects an investigator identity displayed in the clinical trialroster. A user may be permitted to select various attributes associatedwith the selected investigator's (e.g., Olivia Romaney) participation ina clinical trial. For example, the user may select an investigation siteat which the investigator should perform their clinical trial work 1502,a rationale for choosing the investigator to participate in the trial1504, a role that the investigator is expected to play in the trial1506, and a target number of patients that the investigator is expectedto examine for the trial 1508. In some examples, the interface 1500 mayinclude a list 1510 of potential investigation sites for theinvestigator to perform their clinical trial studies. The list 1510 mayinclude sites near the investigator, sites at which the investigator hasadmission privileges, sites preferred by the user, or sites withspecific equipment or facilities required for the trial. In someimplementations, the list may include only sites meeting a user selectedfilter criteria, for example, to limit the listed sites to only thoserelevant to the user's clinical trial.

FIG. 16 is a flow chart of an example process 1600 for generating a userinterface for generating a roster of investigators. In some examples,the example process 1600 can be provided as one or morecomputer-executable programs executed using one or more computingdevices. In some examples, the process 1600 is executed by a computingsystems at an analytical infrastructure. In some examples, the process1600 may enable more efficient use and organization of investigatordata, and thereby, reduce the computing resources required to bothmaintain and process investigator data, and to coordinate logisticsrelated to organizing investigators for clinical trials.

The computing systems at the analytical infrastructure obtain identitiesof a plurality of investigators and data representing a set ofattributes associated with each of the plurality of investigators from adatabase (1602). For example, the identities of a plurality ofinvestigators and data representing a set of attributes associated witheach of the plurality of investigators is obtained from two or more datasets. For example, the identities and associated data can be obtainedfrom a proprietary data set such as, for example, a data set owned by auser of the application (e.g., an organization or business such as apharmaceutical corporation), and from a third-party data set such as,for example, a data set compiled by a third-party data source (e.g., ahealth care information service such as IMS Health). The attributesassociated with each of the plurality of investigators may include, forexample, identifying data for the investigator, key performanceindicator scores, and/or raw data used to calculate key performanceindicator scores. Identifying data for investigators may, for example,include data such as name, contact information, and medical specialty.Raw data may include, for example, historical clinical trial data andhealth records data used to calculate key performance indicator scores.

Additionally, for example, the proprietary data set can include, but isnot limited to, clinical trial data created and maintained by anorganization from previously performed clinical trials. For example,proprietary data can include lists of clinical trial investigators thatthe organization may have previously hired or researched for clinicaltrials. In addition, the proprietary data can include various attributesrelated to the listed investigators, for example, contact information,data from historical clinical trials performed for the organization, andinternally computed key performance indicator scores associated with theinvestigators (e.g., performance metrics generated by the organizationbased on historical clinical trial data from trials performed by theinvestigator for the organization).

Additionally, for example, the third-party data set can include, but isnot limited to, clinical trial data obtained and maintained bythird-party data source based on data related to clinical trialsperformed by multiple organizations, patient data, insurance data,physician data, and data from other appropriate data sources compiled bythe third-party data source. For example, third-party data can includelists of physicians who have conducted or are eligible to conductclinical trials. In some implementations, the third-party physician listincludes all or most of the investigators know to the organization andmany additional investigators (e.g., physicians) eligible to performclinical trials. In addition, the third-party data can include variousattributes related to the physicians, for example, contact information,professional information, key performance indicator scores based on datafrom historical clinical trials performed for multiple organizations,raw data associated with the key performance indicator scores, proximityto patients (e.g., areas of high patient density), number and type ofprocedures performed, clinical trial experience (e.g., a total number oftrials performed by the investigator over a defined period of time),specialties, relevant publications, etc.

In some examples, data from the proprietary data set and the third-partydata set are obtained from separate databases (e.g., a proprietary datadatabase and a third-party database). For example, the application canaccess the proprietary data set from a user organization's database(s),and the third-party data set from one or more third-party database. Forexample, an organization can lease access to the third-party data set,and can be provided access through a user account with the third-party.

The computing systems at the analytical infrastructure receive a userinput indicating a subset of attributes from the set of attributesassociated with each of the plurality of investigators (1604). The userinput may be received from a menu of attributes in a drop down task bar,for example. In some implementations, default attribute selections maybe provided to a user, and the user may be permitted to alter thedefault attribute selections. The user input may, in some examples,include a designation chart dimensions to reflect the attributes.

The computing systems at the analytical infrastructure generate amulti-dimensional chart that organizes the identities of the pluralityof investigators based on the subset of attributes and a userdesignation of selected dimensions to reflect two or more attributesfrom the subset of attributes (1606). The multi-dimensional chart may bea two-dimensional representation of a chart reflecting variousinvestigator attributes along three or more dimensions. The investigatorattributes represented by the dimensions of the multi-dimensional chartmay include the user selected subset of attributes, and, in someexamples, default attributes. For example, the multidimensional chartmay be a bubble chart including a plurality of icons, where each iconrepresents an identity of one of the plurality of investigators. Eachicon may be positioned on the multi-dimensional chart along an x-axisaccording to one user selected attributes of the representedinvestigator and along a y-axis dimension according to another userselected attribute of the represented investigator. A size of each icon(e.g., bubble) may represent a third attribute of the investigatorrepresented by the icon, and a color of the icon may represent a fourthattribute of the investigator. The user input may, in some examples,include a designation chart dimensions to reflect the attributes, andthe computing systems at the analytical infrastructure generate themulti-dimensional chart in accordance with such designations. Forexample, the user input may include a user designation that oneattribute (e.g., a screening rate) is to be represented by a firstdimension of the multi-dimensional chart (e.g., the x-axis) and thatanother attribute (e.g., a screen failure %) is to be represented asecond dimension of the multi-dimensional chart (e.g., the y-axis).

The computing systems at the analytical infrastructure links each of theplurality of icons to a selectable record in a database (1608), andprovides a graphical user interface (GUI) including themulti-dimensional chart and an clinical trial roster for display on acomputing device (1610). The icons may be linked to associated databaserecords so that the user interactions with ones of the plurality oficons cause one or more attributes associated with the ones of theplurality of icons to be altered within the database. For example, thelinking can tie one or more values in either the proprietary data set,the third-party data set, or both to user interactions with therespective icons. For instance, in some implementations because theicons are linked to associated data set records the computing systems atthe analytical infrastructure may modify an attribute associated with aninvestigator identity when the associated icon is added to the roster.In some implementations, the computing systems at the analyticalinfrastructure may modify attributes associated with investigatoridentities added to the roster when the roster is saved.

In some implementations, the modification of the attribute associatedwith an investigator identity may cause a linked icon in anothermulti-dimensional chart (e.g., one being viewed by a different user on aseparate computing device) to be modified based on the modification tothe attribute in one or more data set record(s). In other words, if afirst user drags an icon for investigator Smith into the roster (and, insome examples, saves the roster) that interaction with investigatorSmith's icon may cause one of investigator Smith's attributes to beupdated an applicable data set, for example, an attribute related to thenumber of clinical trials for which investigator Smith is beingconsidered (e.g., a Considered Studies attribute). When this ConsideredStudies attribute for investigator Smith is updated in the applicabledata set(s), the computing systems at the analytical infrastructure maycause an attribute of investigator Smith's icon to change in amultidimensional chart being viewed by a second user. For example, if inthe second user's chart the second user has defined that ConsideredStudies is to be represented by the icon size, the size of investigatorSmith's icon may increase in response to the first user addinginvestigator Smith to the first user's roster. This update to the seconduser's chart may occur in real time (e.g., with minimal delays due toprocessing), on a system directed regular or irregular refresh of thesecond users interface, or a user triggered refresh of the second userinterface.

The computing systems at the analytical infrastructure receives a userselection of one or more icons from the multi-dimensional chart forinclusion in a clinical trial (1612). For instance, the user may bepermitted to drag and drop the icons into the roster. In response to theuser selection, the computing systems at the analytical infrastructureadds identities of investigators represented by the one or more selectedicons to the clinical trial roster (1614). In some implementations, theappearance of icons associated with selected investigators may bemodified to indicate to users that the investigators have already beenadded to the roster. For example, the modification to icons associatedwith investigators added to the roster may include one or more ofshading, highlighting, outlining, hatching, etc. In someimplementations, as mentioned above, adding the icons to the roster maycause data set record(s) that are linked to the icons to be modified.

The computing systems at the analytical infrastructure stores theidentities of investigators represented by the one or more selectedicons in association with the clinical trial roster and themulti-dimensional chart (1616). For example, the user inputs,multi-dimensional chart, and attributes associated with selectedinvestigators may be stored for later editing or as a read-only file toarchive data for later review as needed during the clinical trial. Insome implementations, the stored data may be compared to more recentdata in order to help clinical trial administrators understand changesin investigator performance and/or the original rationale for rosterdecisions.

In some implementations, the process may include an approval step, inwhich a second user (e.g., a supervisor of the user or administrator)must approve a roster generated by the user. Roster approval may, forexample, require authentication of the second user as having appropriatepermissions to approve rosters generally or a roster for a specifictrial. Approval may require the computing systems at the analyticalinfrastructure to validate one or more electronic credentials (e.g., auser name and password, one or more biometric credentials, an electronicID, or other appropriate electronic credential) associated with thesecond user to authenticate the second user, and to establish the seconduser's permissions. In some implementations, the investigatoridentities, multi-dimensional chart, and clinical trial roster are notstored until the roster is approved.

In some implementations, the data set record(s) linked to icons ofinvestigators who have been added to a roster may not be modified untilthe roster is approved. In some implementations, an additional attributeassociated stored in the data set(s) and with investigators who areincluded in the roster may be modified when the roster is approved. Forexample, in such implementations when a user selects an icon associatedwith investigator Smith for inclusion in the roster, the selection ofinvestigator Smith's icon causes one of investigator Smith's attributesto be updated in the appropriate data set(s), for example, an attributerelated to the number of clinical trials for which investigator Smith isbeing considered. Subsequently, when the roster including investigatorSmith is approved another attribute associated with investigator Smithmay be updated in the data set record(s), for example, an attributerelated to the number of active clinical trials for which investigatorSmith is currently included.

In some implementations, a user may be permitted to select one or moreattributes related to an investigator's participation in a clinicaltrial, for example, as described above in reference to FIG. 15. In suchimplementations, a user selection of the one or more attributes may bereceived and stored in associate with the investigator's identity. Userselectable attributes may include, for example, an assignment of aninvestigation site for the examiner to perform clinical trial studies, arationale for choosing the investigator to participate in a clinicaltrial, a role that the investigator is expected to play in a clinicaltrial, and a target number of patients that the investigator is expectedto examine for a clinical trial. In some examples, an attribute relatedto the investigator may be altered in one of the linked data sets inresponse to receiving the user selection. In some examples, the userselections may be stored in association with the investigator's identityin the clinical trial roster.

In some implementations, when a roster is approved electronicnotification (e.g., e-mail, text message, voice message, or otherappropriate electronic message) can be sent to one or more of theinvestigators included in the roster using contact information for eachinvestigator from the database. In some implementations, the electronicnotification may require an investigator to confirm their availabilityfor the trial and upon receiving the confirmation one or more attributesassociated with the respective investigator may be updated in thedatabase, for example, an attribute related to active clinical trials inwhich investigator is participating. In addition, the notification mayinclude information related to an investigator's participation in thestudy, for example, an investigation site to which the investigator hasbeen assigned for performing the clinical trial (e.g., a hospital,clinic, university, or other medical facility), or a target number ofparticipants for the investigator's portion of the clinical trial. Insome implementations, the notification may include user selectableinputs allowing an investigator to accept, decline, or suggest changesto information related to the investigator's participation in theclinical trial. For example, the investigator may be permitted to acceptor decline participation in the clinical trial, and to suggest changesto the investigation site, target number of participants, or othercriteria related to the investigator's participation in the clinicaltrial.

In some implementations, the computing systems at the analyticalinfrastructure may select a subset of identities from the plurality ofinvestigators for inclusion in the multi-dimensional chart based on oneor more user-selected filtering criteria. The user selected filteringcriteria may include some or all of the attributes associated with theinvestigators. In some implementations, the computing systems at theanalytical infrastructure may permit a user to filter investigator databased on multiple filtering levels. In some implementations, thecomputing systems at the analytical infrastructure may permit a user toselect filtering criteria based on the data set to which the filteringcriteria data is related. For example, a primary filtering criteria maybe investigators included in the proprietary data set, and a secondaryfiltering criteria may include investigators included in the third-partydata set. Thus, a user can identify overlapping data between theproprietary data set and the third-party data set.

In some implementations, a user may submit a previously assembled listof investigators (e.g., a previously generated clinical trial roster),and the computing systems at the analytical infrastructure may generatethe multi-dimensional chart based on the investigators included in thesubmitted list. The previously assembled list may be a stored clinicaltrial roster (e.g., a draft roster or a roster from a different clinicaltrial). In some implementations, the roster may be persistent acrossvarious user interfaces. For example, a user may begin generating aroster while viewing the heat chart depicted in interface 900 of FIG. 9and desire to further refine the roster based on the multi-dimensionalchart depicted in interface 1100 or FIG. 11. In such an implementation,when the user choses to switch from interface 900 to interface 1100, thecomputing systems at the analytical infrastructure may generate themulti-dimensional chart based on the investigators included in theroster from interface 900. Some implementations may include a userselectable option to make the roster persistent or not persistentbetween multiple interfaces.

In some implementations, icons associated with investigators who havenot yet performed a clinical trial or for whom no historical clinicaltrial data is available may be given a distinctive appearance (e.g., adistinctive color or shape). For example, icons associated with suchinvestigators may be shaded grey.

Enabling user manipulation of large amounts of data from multiple datasets and relating disparate factors to identify extraordinary objectscan require large amounts of memory and processing cycles. Consideringeach dimension of a database individually may reveal only modestdifferences between relative targets. Yet as disparate factors arerelated to one another and as a user is allowed to change analyticcriteria in real time across large volumes of data, an administrator maybe able to perceive a degree of persistency of desirable characteristicswhile also recognizing the relative suitability, oftentimes diminished,of other targets as the criteria and dimensions are modified. Thispersistency may become even more compelling when three or fourdimensions of consistency are considered and an administrator is allowedto perceive targets of interest whose ordinal metric, under newcriteria, may not surface into a display of top targets but for a userdesignation to maintain selected objects within a data view of legacytargets under the new criteria. While advances in computer technologyhave greatly increased the amount of available information, the sheervolume of information can be overwhelming and cumbersome to the extentthat processors may struggle to operate on data sets in time such that auser can perceive the impact of new criteria in real-time. In someconfigurations, real-time is defined as the time required to maintain aTCP connection across a wide area network. In other configurations,real-time is defined as the ability to render a new display within athreshold degree of time (e.g., 1 second, 3 seconds, or 10 seconds). Byconfiguring the database to perform preprocessing in a way thatfacilitates real-time updates to a display, the user is provided with aninvestigative tool that allows multidimensional target investigation ina manner capable of allowing a user to perceive the impact of aparticular factor on relative performance.

Implementations of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly-implemented computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Implementations of the subject matter described inthis specification can be implemented as one or more computer programs,i.e., one or more modules of computer program instructions encoded on atangible non-transitory program carrier for execution by, or to controlthe operation of, data processing apparatus. The computer storage mediumcan be a machine-readable storage device, a machine-readable storagesubstrate, a random or serial access memory device, or a combination ofone or more of them.

The term “data processing apparatus” refers to data processing hardwareand encompasses all kinds of apparatus, devices, and machines forprocessing data, including, by way of example, a programmable processor,a computer, or multiple processors or computers. The apparatus can alsobe or further include special purpose logic circuitry, e.g., a centralprocessing unit (CPU), a FPGA (field programmable gate array), or anASIC (application-specific integrated circuit). In some implementations,the data processing apparatus and/or special purpose logic circuitry maybe hardware-based and/or software-based. The apparatus can optionallyinclude code that creates an execution environment for computerprograms, e.g., code that constitutes processor firmware, a protocolstack, a database management system, an operating system, or acombination of one or more of them. The present disclosure contemplatesthe use of data processing apparatuses with or without conventionaloperating systems, for example Linux, UNIX, Windows, Mac OS, Android,iOS or any other suitable conventional operating system.

A computer program, which may also be referred to or described as aprogram, software, a software application, a module, a software module,a script, or code, can be written in any form of programming language,including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program may, butneed not, correspond to a file in a file system. A program can be storedin a portion of a file that holds other programs or data, e.g., one ormore scripts stored in a markup language document, in a single filededicated to the program in question, or in multiple coordinated files,e.g., files that store one or more modules, sub-programs, or portions ofcode. A computer program can be deployed to be executed on one computeror on multiple computers that are located at one site or distributedacross multiple sites and interconnected by a communication network.While portions of the programs illustrated in the various figures areshown as individual modules that implement the various features andfunctionality through various objects, methods, or other processes, theprograms may instead include a number of sub-modules, third partyservices, components, libraries, and such, as appropriate. Conversely,the features and functionality of various components can be combinedinto single components as appropriate.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., a central processing unit (CPU), a FPGA (fieldprogrammable gate array), or an ASIC (application-specific integratedcircuit).

Computers suitable for the execution of a computer program include, byway of example, can be based on general or special purposemicroprocessors or both, or any other kind of central processing unit.Generally, a central processing unit will receive instructions and datafrom a read-only memory or a random access memory or both. The essentialelements of a computer are a central processing unit for performing orexecuting instructions and one or more memory devices for storinginstructions and data. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto-optical disks, or optical disks. However, a computer need nothave such devices. Moreover, a computer can be embedded in anotherdevice, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio or video player, a game console, a Global PositioningSystem (GPS) receiver, or a portable storage device, e.g., a universalserial bus (USB) flash drive, to name just a few.

Computer-readable media (transitory or non-transitory, as appropriate)suitable for storing computer program instructions and data include allforms of non-volatile memory, media and memory devices, including by wayof example semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The memorymay store various objects or data, including caches, classes,frameworks, applications, backup data, jobs, web pages, web pagetemplates, database tables, repositories storing business and/or dynamicinformation, and any other appropriate information including anyparameters, variables, algorithms, instructions, rules, constraints, orreferences thereto. Additionally, the memory may include any otherappropriate data, such as logs, policies, security or access data,reporting files, as well as others. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube), LCD (liquidcrystal display), or plasma monitor, for displaying information to theuser and a keyboard and a pointing device, e.g., a mouse or a trackball,by which the user can provide input to the computer. Other kinds ofdevices can be used to provide for interaction with a user as well; forexample, feedback provided to the user can be any form of sensoryfeedback, e.g., visual feedback, auditory feedback, or tactile feedback;and input from the user can be received in any form, including acoustic,speech, or tactile input. In addition, a computer can interact with auser by sending documents to and receiving documents from a device thatis used by the user; for example, by sending web pages to a web browseron a user's client device in response to requests received from the webbrowser.

The term “graphical user interface,” or GUI, may be used in the singularor the plural to describe one or more graphical user interfaces and eachof the displays of a particular graphical user interface. Therefore, aGUI may represent any graphical user interface, including but notlimited to, a web browser, a touch screen, or a command line interface(CLI) that processes information and efficiently presents theinformation results to the user. In general, a GUI may include aplurality of user interface (UI) elements, some or all associated with aweb browser, such as interactive fields, pull-down lists, and buttonsoperable by the business suite user. These and other UI elements may berelated to or represent the functions of the web browser.

Implementations of the subject matter described in this specificationcan be implemented in a computing system that includes a back-endcomponent, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front-endcomponent, e.g., a client computer having a graphical user interface ora Web browser through which a user can interact with an implementationof the subject matter described in this specification, or anycombination of one or more such back-end, middleware, or front-endcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, e.g., a communicationnetwork. Examples of communication networks include a local area network(LAN), a wide area network (WAN), e.g., the Internet, and a wirelesslocal area network (WLAN).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or on the scope of what may be claimed, but rather asdescriptions of features that may be specific to particularimplementations of particular inventions. Certain features that aredescribed in this specification in the context of separateimplementations can also be implemented in combination in a singleimplementation. Conversely, various features that are described in thecontext of a single implementation can also be implemented in multipleimplementations separately or in any suitable sub-combination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asub-combination or variation of sub-combinations.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be helpful. Moreover, the separation of various system modules andcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Particular implementations of the subject matter have been described.Other implementations, alterations, and permutations of the describedimplementations are within the scope of the following claims as will beapparent to those skilled in the art. For example, the actions recitedin the claims can be performed in a different order and still achievedesirable results.

Accordingly, the above description of example implementations does notdefine or constrain this disclosure. Other changes, substitutions, andalterations are also possible without departing from the spirit andscope of this disclosure.

The invention claimed is:
 1. A computer-implemented method fororganizing clinical trial data executed by one or more processors, themethod comprising: obtaining, by the one or more processors of a serversystem and from a selectable record in an aggregate database of theserver system, identities of a plurality of investigators and datarepresenting a set of attributes associated with each of the pluralityof investigators from a first data set and a second data set, wherein:the first data set containing proprietary data associated with at leastone of the investigators and received from a first set of databases, thesecond data set containing third-party data associated with at least oneof the investigators and received from a second set of databases that isdifferent from the first set of databases, and the selectable recordenables the one or more processors to perform one or more adjustments todata of the identities of the plurality of investigators included withinthe aggregate database in a first time period that is shorter than asecond time period for performing the one or more adjustments on data ofthe identities of the plurality of investigators included within thefirst set of databases and the second set of databases but not storedwithin the aggregate database; receiving, by the one or more processorsand from a computing device, a user input indicating a subset ofattributes from the set of attributes associated with each of theplurality of investigators; generating, by the one or more processors, amulti-dimensional chart that organizes the identities of the pluralityof investigators based on the subset of attributes and a userdesignation of selected dimensions to reflect two or more of attributesfrom the subset of attributes, the multi-dimensional chart comprising: afirst dimension representing a first attribute from the subset ofattributes; a second dimension representing a second attribute from thesubset of attributes; and a plurality of icons, each icon representingan identity of one of the plurality of investigators, wherein each iconis positioned on the multidimensional chart along the first dimensionaccording to a value of the first attribute associated with therepresented identity and along the second dimension according to a valueof the second attribute of the represented identity, and wherein agraphical property of each icon represents a value of a third attributeof the represented identity; linking, by the one or more processors,each icon included in the plurality of icons to the selectable record inthe aggregate database so that user interactions with icons included inthe plurality of icons by the computing device cause one or moreattributes associated with the icons included in the plurality of iconsto be altered within the aggregate database; providing, by the one ormore processors and for display on the computing device, a graphicaluser interface (GUI) including the multi-dimensional chart and aclinical trial roster; receiving, by the one or more processors and fromthe computing device, a user selection of one or more icons from amongthe plurality of icons for inclusion in a clinical trial; in response toreceiving the user selection: adding, by the one or more processors,identities of investigators represented by the one or more selectedicons to the clinical trial roster; updating, by the one or moreprocessors, the selectable record to reflect that the identities ofinvestigators represented by the one or more selected icons have beenadded to the clinical trial roster; and updating, by the one or moreprocessors and based on linking each icon included in the plurality oficons to the selectable record in the aggregate database, one or moreattributes in the selectable record that are associated with the one ormore selected icons.
 2. The method of claim 1, comprising: in responseto adding the identities of investigators represented by the one or moreselected icons to the clinical trial roster, modifying, by the one ormore processors, an attribute associated with at least one of the addedidentities; and causing, by the one or more processors, a linked icon inanother multi-dimensional chart to be modified based on the attributeassociated with the at least one of the added identities being modified.3. The method of claim 1, comprising selecting, by the one or moreprocessors and based on user selected filtering criteria, a subset ofidentities of the plurality of investigators, and wherein generating themulti-dimensional chart comprises generating the multi-dimensional chartto organize the identities in the subset of identities of the pluralityof investigators, and wherein each icon of the plurality of iconsrepresents an identity from the subset of identities of the plurality ofinvestigators.
 4. The method of claim 1, comprising receiving, by theone or more processors, an approval indication for the clinical trialroster, and wherein the selectable record to reflect that the identitiesof investigators represented by the one or more selected icons have beenadded to the clinical trial roster is updated in response to receivingthe approval indication.
 5. The method of claim 1, wherein the userinput indicating a subset of attributes comprises the user designation,and the user designation indicates to represent the first attribute bythe first dimension of the multi-dimensional chart and the secondattribute by the second dimension of the multi-dimensional chart.
 6. Themethod of claim 1, wherein the subset of attributes includes the thirdattribute, and wherein the user input indicating a subset of attributescomprises the user designation, and the user designation indicates torepresent the first attribute by the first dimension of themulti-dimensional chart, the second attribute by the second dimension ofthe multi-dimensional chart, and the third attribute by a size of eachicon of the multi-dimensional chart.
 7. The method of claim 1, whereinthe data representing the set of attributes associated with each of theplurality of investigators comprises raw data and performance indicatorscores.
 8. The method of claim 1, wherein a color of each iconrepresents a fourth attribute of the represented investigator.
 9. Themethod of claim 1, comprising altering, by the one or more processors,icons associated with the identities of investigators that have beenadded to the clinical trial roster.
 10. The method of claim 1,comprising providing, by the one or more processors and for display onthe computing device, a list of attributes associated with an identityof an investigator when a user selection device hovers over an icon thatrepresents the identity of the investigator.
 11. A system comprising:obtaining, by one or more processors of a server system and from aselectable record in an aggregate database of the server system,identities of a plurality of investigators and data representing a setof attributes associated with each of the plurality of investigatorsfrom a first data set and a second data set, wherein: the first data setcontaining proprietary data associated with at least one of theinvestigators and received from a first set of databases, the seconddata set containing third-party data associated with at least one of theinvestigators and received from a second set of databases that isdifferent from the first set of databases, and the selectable recordenables the one or more processors to perform one or more adjustments todata of the identities of the plurality of investigators included withinthe aggregate database in a first time period that is shorter than asecond time period for performing the one or more adjustments on data ofthe identities of the plurality of investigators included within thefirst set of databases and the second set of databases but not storedwithin the aggregate database; receiving, by the one or more processorsand from a computing device, a user input indicating a subset ofattributes from the set of attributes associated with each of theplurality of investigators; generating, by the one or more processors, amulti-dimensional chart that organizes the identities of the pluralityof investigators based on the subset of attributes and a userdesignation of selected dimensions to reflect two or more of attributesfrom the subset of attributes, the multi-dimensional chart comprising: afirst dimension representing a first attribute from the subset ofattributes; a second dimension representing a second attribute from thesubset of attributes; and a plurality of icons, each icon representingan identity of one of the plurality of investigators, wherein each iconis positioned on the multi-dimensional chart along the first dimensionaccording to a value of the first attribute associated with therepresented identity and along the second dimension according to a valueof the second attribute of the represented identity, and wherein agraphical property of each icon represents a value of a third attributeof the represented identity; linking, by the one or more processors,each icon included in the plurality of icons to the selectable record inthe aggregate database so that user interactions with icons included inthe plurality of icons by the computing device cause one or moreattributes associated with the icons included in the plurality of iconsto be altered within the aggregate database; providing, by the one ormore processors and for display on the computing device, a graphicaluser interface (GUI) including the multi-dimensional chart and aclinical trial roster; receiving, by the one or more processors and fromthe computing device, a user selection of one or more icons from amongthe plurality of icons for inclusion in a clinical trial; in response toreceiving the user selection: adding, by the one or more processors,identities of investigators represented by the one or more selectedicons to the clinical trial roster; updating, by the one or moreprocessors, the selectable record to reflect that the identities ofinvestigators represented by the one or more selected icons have beenadded to the clinical trial roster; and updating, by the one or moreprocessors and based on linking each icon included in the plurality oficons to the selectable record in the aggregate database, one or moreattributes in the selectable record that are associated with the one ormore selected icons.
 12. The system of claim 11, comprising instructionswhich cause the one or more processors to perform operations comprising:in response to adding the identities of investigators represented by theone or more selected icons to the clinical trial roster, modifying, bythe one or more processors, an attribute associated with at least one ofthe added identities; and causing, by the one or more processors, alinked icon in another multi-dimensional chart to be modified based onthe attribute associated with the at least one of the added identitiesbeing modified.
 13. The system of claim 11, comprising instructionswhich cause the one or more processors to perform operations comprising:selecting, by the one or more processors and based on user selectedfiltering criteria, a subset of identities of the plurality ofinvestigators, and wherein generating the multi-dimensional chartcomprises generating the multi-dimensional chart to organize theidentities in the subset of identities of the plurality ofinvestigators, and wherein each icon of the plurality of iconsrepresents an identity from the subset of identities of the plurality ofinvestigators.
 14. The system of claim 11, comprising instructions whichcause the one or more processors to perform operations comprising:receiving, by the one or more processors, an approval indication for theclinical trial roster, and wherein the selectable record to reflect thatthe identities of investigators represented by the one or more selectedicons have been added to the clinical trial roster is updated inresponse to receiving the approval indication.
 15. The system of claim11, wherein the user input indicating a subset of attributes comprisesthe user designation, and the user designation indicates to representthe first attribute by the first dimension of the multi-dimensionalchart and the second attribute by the second dimension of themulti-dimensional chart.
 16. A non-transient computer readable mediumstoring instructions that, when executed by one or more processors,cause the one or more processors to perform operations comprising:obtaining, by the one or more processors of a server system and from aselectable record in an aggregate database of the server system,identities of a plurality of investigators and data representing a setof attributes associated with each of the plurality of investigatorsfrom a first data set and a second data set, wherein: the first data setcontaining proprietary data associated with at least one of theinvestigators and received from a first set of databases, the seconddata set containing third-party data associated with at least one of theinvestigators and received from a second set of databases that isdifferent from the first set of databases, and the selectable recordenables the one or more processors to perform one or more adjustments todata of the identities of the plurality of investigators included withinthe aggregate database in a first time period that is shorter than asecond time period for performing the one or more adjustments on data ofthe identities of the plurality of investigators included within thefirst set of databases and the second set of databases but not storedwithin the aggregate database; receiving, by the one or more processorsand from a computing device, a user input indicating a subset ofattributes from the set of attributes associated with each of theplurality of investigators; generating, by the one or more processors, amulti-dimensional chart that organizes the identities of the pluralityof investigators based on the subset of attributes and a userdesignation of selected dimensions to reflect two or more of attributesfrom the subset of attributes, the multi-dimensional chart comprising: afirst dimension representing a first attribute from the subset ofattributes; a second dimension representing a second attribute from thesubset of attributes; and a plurality of icons, each icon representingan identity of one of the plurality of investigators, wherein each iconis positioned on the multi-dimensional chart along the first dimensionaccording to a value of the first attribute associated with therepresented identity and along the second dimension according to a valueof the second attribute of the represented identity, and wherein agraphical property of each icon represents a value of a third attributeof the represented identity; linking, by the one or more processors,each icon included in the plurality of icons to the selectable record inthe aggregate database so that user interactions with icons included inthe plurality of icons by the computing device cause one or moreattributes associated with the icons included in the plurality of iconsto be altered within the aggregate database; providing, by the one ormore processors and for display on the computing device, a graphicaluser interface (GUI) including the multi-dimensional chart and aclinical trial roster; receiving, by the one or more processors and fromthe computing device, a user selection of one or more icons from amongthe plurality of icons for inclusion in a clinical trial; in response toreceiving the user selection: adding, by the one or more processors,identities of investigators represented by the one or more selectedicons to the clinical trial roster; updating, by the one or moreprocessors, the selectable record to reflect that the identities ofinvestigators represented by the one or more selected icons have beenadded to the clinical trial roster; and updating, by the one or moreprocessors and based on linking each icon included in the plurality oficons to the selectable record in the aggregate database, one or moreattributes in the selectable record that are associated with the one ormore selected icons.
 17. The medium of claim 16, comprising: in responseto adding the identities of investigators represented by the one or moreselected icons to the clinical trial roster, modifying, by the one ormore processors, an attribute associated with at least one of the addedidentities; and causing, by the one or more processors, a linked icon inanother multi-dimensional chart to be modified based on the attributeassociated with the at least one of the added identities being modified.18. The medium of claim 16, comprising selecting, by the one or moreprocessors and based on user selected filtering criteria, a subset ofidentities of the plurality of investigators, and wherein generating themulti-dimensional chart comprises generating the multi-dimensional chartto organize the identities in the subset of identities of the pluralityof investigators, and wherein each icon of the plurality of iconsrepresents an identity from the subset of identities of the plurality ofinvestigators.
 19. The medium of claim 16, comprising receiving, by theone or more processors, an approval indication for the clinical trialroster, and wherein the selectable record to reflect that the identitiesof investigators represented by the one or more selected icons have beenadded to the clinical trial roster is updated in response to receivingthe approval indication.
 20. The method of claim 1, wherein the userinput indicating a subset of attributes comprises the user designation,and the user designation indicates to represent the first attribute bythe first dimension of the multi-dimensional chart and the secondattribute by the second dimension of the multi-dimensional chart.